Method and apparatus with speech recognition

ABSTRACT

A processor-implemented decoding method in a first neural network is provided. The method predicts probabilities of candidates of an output token based on at least one previously input token, determines the output token among the candidates based on the predicted probabilities; and determines a next input token by selecting one of the output token and a pre-defined special token based on a determined probability of the output token.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 USC § 119(a) of KoreanPatent Application No. 10-2018-0139787 filed on Nov. 14, 2018 in theKorean Intellectual Property Office, the entire disclosure of which isincorporated herein by reference for all purposes.

BACKGROUND 1. Field

This application relates to a method and apparatus with speechrecognition.

2. Description of Related Art

When a sequence with an undefined length is received as an input in anartificial neural network, an output of an undefined length may begenerated. Typically, to address this problem, an encoder-decoderartificial neural network model, which is a type of sequence-to-sequencemodel, may be implemented. For example, a method of continuouslyoutputting a subsequent output for each token corresponding to a unitconstituting one sequence, based on a previous output of the neuralnetwork as an input, in order to calculate the output with the undefinedlength using the artificial neural network may be referred to as anauto-regressive decoding.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In a general aspect, a processor-implemented decoding method in a firstneural network comprises predicting probabilities of candidates of anoutput token based on at least one previously input token, determiningthe output token among the candidates based on the predictedprobabilities, and determining a next input token by selecting one ofthe output token and a pre-defined special token based on a determinedprobability of the output token.

The special token may be determined to be the next input token, areliability of candidates of a next output token predicted based on thespecial token is higher than a reliability of candidates of a nextoutput token predicted based on the output token.

The determining of the output token may include selecting the outputtoken from the candidates based on a combination of probabilities of thecandidates predicted by one or more second neural networks interworkingwith the first artificial neural network, and probabilities of thecandidates predicted by the first neural network.

An input layer of the neural network may include a plurality of nodescorresponding to each of the candidates, and a special nodecorresponding to the special token.

The at least one previously input token may include at least one of anoutput token previously predicted by the neural network and the specialtoken.

The determining of the output token may include selecting a token havinga highest probability among the candidates.

The method may be a recognition method using the first and the one ormore second neural networks.

The determining of the next input token may include comparing aprobability of the output token to a threshold probability, selectingthe special token as the next input token when the probability of theoutput token is lower than the threshold probability, and selecting theoutput token as the next input token when the probability of the outputtoken is higher than or equal to the threshold probability.

The predicting of the probabilities of the candidates of the outputtoken may include predicting probabilities of the candidates of theoutput token based on a relationship between the at least one previouslyinput token and the output token.

The method may further include determining a next output token based onthe at least one previously input token and the next input token.

The method may further include masking the special token to preventtokens other than the special token from attending to the special token.

The method may further include setting a relationship between the nextoutput token and the special token to be less than or equal to apredetermined value when the special token is determined to be the nextinput token.

The determining of the output token may include determining whether theoutput token is similar to the special token, and determining a tokenhaving a second highest probability among the candidates to be theoutput token when the output token is similar to the special token.

In a general aspect, a processor-implemented decoding method in a firstneural network includes predicting probabilities of candidates of anoutput token based on at least one previously input token, selecting theoutput token from the candidates based on a combination of probabilitiesof candidates predicted by one or more second neural networksinterworking with the first neural network and probabilities ofcandidates predicted by the first neural network, comparing aprobability of the output token in the first neural network and aprobability of the output token in a determined neural network among theone or more second neural networks which has a highest probability ofthe output token, and determining a next input token in the first neuralnetwork by selecting one of the output token and a pre-defined specialtoken based on a result of the comparing.

The determining of the next input token may include selecting thespecial token to be the next input token of the first neural networkwhen a difference between the probability of the output token in thefirst neural network and the probability of the output token in thedetermined neural network is greater than or equal to a threshold value,and selecting the output token to be the next input token of the firstneural network when a difference between the probability of the outputtoken in the first neural network and the probability of the outputtoken in the determined neural network is less than the threshold value.

The difference between the probability of the output token in the firstneural network and the probability of the output token in the determinedneural network is greater than or equal to the threshold value, areliability of candidates of a next output token predicted based on thespecial token is higher than a reliability of candidates of a nextoutput token predicted by the output token.

An input layer of the neural network may include a plurality of nodescorresponding to each of the candidates, and a special nodecorresponding to the special token.

The at least one previously input token may include at least one of anoutput token previously predicted by the artificial neural network andthe special token.

The predicting of the probabilities of the candidates of the outputtoken may include predicting probabilities of the candidates of theoutput token based on a relationship between the at least one previouslyinput token and the output token.

The method may include determining a next output token based on the atleast one previously input token and the next input token.

The method may include masking the special token to prevent tokens otherthan the special token from attending to the special token.

The method may include setting a relationship between the next outputtoken and the special token to be less than or equal to a predeterminedvalue when the special token is determined to be the next input token.

The determining of the output token may include determining whether theoutput token is similar to the special token, and determining a tokenhaving a second highest probability among the candidates to be theoutput token when the output token is similar to the special token.

In a general aspect, a speech recognition apparatus includes one or moreprocessors configured to implement at least a first neural networkincluding the one or more processors configured to predict probabilitiesof candidates of an output token based on at least one previously inputtoken input, determine the output token among the candidates based onthe predicted probabilities, and determine a next input token byselecting one of the output token and a pre-defined special token basedon a determined probability of the output token.

When the special token is determined to be the next input token, areliability of candidates of a next output token predicted based on thespecial token may be higher than a reliability of candidates of a nextoutput token predicted based on the output token.

An input layer of the neural network may include a plurality of nodescorresponding to each of the candidates, and a special nodecorresponding to the special token.

The at least one previously input token may include at least one of anoutput token previously predicted by the neural network and the specialtoken.

The one or more processors may be configured to select a token having ahighest probability among the candidates.

The one or more processors may be further configured to implement one ormore second neural networks, and wherein the one or more processors arefurther configured to select the output token from the candidates basedon a combination of probabilities of the candidates predicted by asecond neural network interworking with the first neural network, andprobabilities of the candidates predicted by the first neural network.

The one or more processors may be further configured to compare aprobability of the output token to a threshold probability, select thespecial token as the next input token when the probability of the outputtoken is lower than the threshold probability, and select the outputtoken as the next input token when the probability of the output tokenis higher than or equal to the threshold probability.

The one or more processors may be further configured to predictprobabilities of the candidates of the output token based on arelationship between the at least one previously input token and theoutput token.

The one or more processors may be further configured to determine a nextoutput token based on the at least one previously input token and thenext input token.

The one or more processors may be further configured to mask the specialtoken to prevent tokens other than the special token from attending tothe special token.

The one or more processors may be further configured to set arelationship between the next output token and the special token to beless than or equal to a predetermined value when the special token isdetermined to be the next input token.

The one or more processors may be further configured to determinewhether the output token is similar to the special token, and determinea token having a second highest probability among the candidates to bethe output token when the output token is similar to the special token.

In a general aspect, a speech recognition apparatus includes one or moreprocessors configured to implement at least a first neural networkincluding the processor configured to predict probabilities ofcandidates of an output token based on at least one previously inputtoken, select the output token from the candidates based on acombination of probabilities of the candidates predicted by one or moresecond neural networks interworking with the first neural network andprobabilities of the candidates predicted by the first neural network,compare a probability of the output token in the first neural networkand a probability of the output token in a determined neural networkamong the one or more second neural networks which has a highestprobability of the output token, and determine a next input token in thefirst neural network by selecting one of the output token and apre-defined special token based on a result of the comparing.

The one or more processors may be further configured to select thespecial token to be the next input token of the first neural networkwhen a difference between the probability of the output token in thefirst neural network and the probability of the output token in thedetermined neural network is greater than or equal to a threshold value,and select the output token to be the next input token of the firstneural network when a difference between the probability of the outputtoken in the first neural network and the probability of the outputtoken in the determined neural network is less than the threshold value.

When a difference between the probability of the output token in thefirst neural network and the probability of the output token in thedetermined neural network is greater than or equal to the thresholdvalue, a reliability of candidates of a next output token predictedbased on the special token is higher than a reliability of candidates ofa next output token predicted by the output token.

In a general aspect, a speech recognition system includes one or moreprocessors, one or more memories, storing instructions that, whenexecuted by the one or more processors, configured the one or moreprocessors to extract a speech feature from an input speech signal andgenerate an encoded feature, determine probabilities of candidates of anoutput token based on the encoded feature and a previously determinedrecognition result, and determine a next output token based on adetermined weight of candidates generated by a language model neuralnetwork, and a determined weight of candidates determined by a speechrecognition neural network.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example of an auto-regressive decoding method inan artificial neural network, in accordance with one or moreembodiments;

FIG. 2 illustrates an example of a method of masking a special token inan artificial neural network to which an attention mechanism is appliedin accordance with one or more embodiments;

FIG. 3 illustrates an example of an encoder-decoder structure artificialneural network system, in accordance with one or more embodiments;

FIG. 4 illustrates an example of an encoder-decoder structure artificialneural network system based on an ensemble technique in accordance withone or more embodiments;

FIG. 5 illustrates an example of an application in a speech recognitionartificial neural network having an additional language model inaccordance with one or more embodiments;

FIG. 6 illustrates an example of a speech recognition artificial neuralnetwork system having an additional language model artificial neuralnetwork in accordance with one or more embodiments;

FIG. 7A illustrates an example of applying an ensemble technique to alanguage model artificial neural network decoder and a speechrecognition artificial neural network decoder in accordance with one ormore embodiments;

FIG. 7B illustrates an example of applying an ensemble technique to aspeech recognition artificial neural network decoder and to a languagemodel artificial neural network decoder using a special token inaccordance with one or more embodiments;

FIG. 8 illustrates an example of a decoding method in an artificialneural network in accordance with one or more embodiments;

FIG. 9 illustrates an example of a method of processing a token with alow probability in an auto-regressive-based sequence generatingartificial neural network in accordance with one or more embodiments;and

FIG. 10 illustrates an example of an apparatus processing a token with alow probability in an auto-regressive-based sequence generatingartificial neural network.

Throughout the drawings and the detailed description, unless otherwisedescribed or provided, the same drawing reference numerals will beunderstood to refer to the same elements, features, and structures. Thedrawings may not be to scale, and the relative size, proportions, anddepiction of elements in the drawings may be exaggerated for clarity,illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, apparatuses,and/or systems described herein. However, various changes,modifications, and equivalents of the methods, apparatuses, and/orsystems described herein will be apparent after an understanding of thedisclosure of this application. For example, the sequences of operationsdescribed herein are merely examples, and are not limited to those setforth herein, but may be changed as will be apparent after anunderstanding of the disclosure of this application, with the exceptionof operations necessarily occurring in a certain order. Also,descriptions of features that are known in the art may be omitted forincreased clarity and conciseness.

The features described herein may be embodied in different forms, andare not to be construed as being limited to the examples describedherein. Rather, the examples described herein have been provided merelyto illustrate some of the many possible ways of implementing themethods, apparatuses, and/or systems described herein that will beapparent after an understanding of the disclosure of this application.

Although terms such as “first,” “second,” and “third” may be used hereinto describe various members, components, regions, layers, or sections,these members, components, regions, layers, or sections are not to belimited by these terms. Rather, these terms are only used to distinguishone member, component, region, layer, or section from another member,component, region, layer, or section. Thus, a first member, component,region, layer, or section referred to in examples described herein mayalso be referred to as a second member, component, region, layer, orsection without departing from the teachings of the examples.

Throughout the specification, when an element, such as a layer, region,or substrate, is described as being “on,” “connected to,” or “coupledto” another element, it may be directly “on,” “connected to,” or“coupled to” the other element, or there may be one or more otherelements intervening therebetween. In contrast, when an element isdescribed as being “directly on,” “directly connected to,” or “directlycoupled to” another element, there can be no other elements interveningtherebetween.

The terminology used herein is for describing various examples only, andis not to be used to limit the disclosure. The articles “a,” “an,” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. The terms “comprises,” “includes,”and “has” specify the presence of stated features, numbers, operations,members, elements, and/or combinations thereof, but do not preclude thepresence or addition of one or more other features, numbers, operations,members, elements, and/or combinations thereof.

Unless otherwise defined, all terms used herein, including technical andscientific terms, used herein have the same meaning as commonlyunderstood by one of ordinary skill in the art to which this disclosurepertains after an understanding of the present disclosure. Terms, suchas those defined in commonly used dictionaries, are to be interpreted ashaving a meaning that is consistent with their meaning in the context ofthe relevant art and the present disclosure, and are not to beinterpreted in an idealized or overly formal sense unless expressly sodefined herein.

Hereinafter, example embodiments will be described in detail withreference to the accompanying drawings. Like numbers refer to likeelements throughout.

FIG. 1 is a diagram illustrating an example of an auto-regressivedecoding method in an artificial neural network.

Referring to FIG. 1, a decoder 120 receives an encoded feature 110 as aninput. For example, the decoder 120 receives an input, e.g., from anexample encoder in an embodiment, connected to a front end of thedecoder 120 in an artificial neural network.

The encoder and the decoder 120 may be a sequence-to-sequenceencoder-decoder implemented by an encoder-decoder neural network. Aneural network may be a deep neural network (DNN), as a non-limitingexample. In such an example, the DNN may include one or more of a fullyconnected network, a deep convolutional network, a recurrent neuralnetwork (RNN), a recurrent deep neural network (RDNN), and/or abidirectional recurrent neural network (BDRNN) may include different oroverlapping neural network portions respectively with such full,convolutional, recurrent connections, and/or bidirectional recurrentconnections. Nodes of layers in the neural network may non-linearlyaffect each another. Also, parameters of the neural network such asvalues output from each of the nodes, connectional relationships betweenthe nodes, and similar parameters may be optimized through learning,e.g., through loss-based back propagation.

The neural networks may be processor implemented neural network models,and various processes may be implemented through the neural networkmodels as specialized computational architectures, which aftersubstantial training may provide computationally intuitive mappingsbetween input patterns and output patterns or pattern recognitions ofinput patterns, as non-limiting examples. The trained capability ofgenerating such mappings or performing such example pattern recognitionsmay be referred to as a learning capability of the neural network. Suchtrained capabilities may also enable the specialized computationalarchitecture to classify such an input pattern, or portion of the inputpattern, as a member that belongs to one or more predetermined groups.Further, because of the specialized training, such specially trainedneural network may thereby have a generalization capability ofgenerating a relatively accurate or reliable output with respect to aninput pattern that the neural network may not have been trained for, forexample.

In an example, the sequence-to-sequence encoder-decoder may have anetwork structure in which an encoder and a decoder are integrated, andmay generate a sequence of recognition results from an input sequence.For example, the encoder and the decoder 120 implemented by thesequence-to-sequence structure may generate recognition resultscorresponding to an input speech from the input speech. The exampleintegrated encoder and the decoder 120 may be pre-trained to generate asequence of recognition results from an input sequence, e.g., initiallytrained prior to the receipt of the input speech.

The use of the term “may” herein with respect to an example orembodiment, e.g., as to what an example or embodiment may include orimplement, means that at least one example or embodiment exists wheresuch a feature is included or implemented while all examples andembodiments are not limited thereto.

The encoder generates the encoded feature 110 by encoding the inputsequence. The encoder generates encoded information by extracting afeature from the input sequence. The encoded feature 110 is applied tothe decoder 120. The decoder 120 generates a recognition result based onthe encoded feature 110.

Referring to FIG. 1, the decoder 120, having received the encodedfeature 110, determines a token1 105 to be an output token based on astart token 115 corresponding to an input token. The decoder 120, havingdetermined the token1 105 as the output token, determines the token1 105to be a next input token.

In the encoder-decoder structure artificial neural network, the decoder120 acquires an output token based on information calculated by theencoder for each step. In this example, the decoder 120 acquires theoutput token depending on input tokens selected up to a previous step.

For example, the decoder 120 having determined the token1 105 as theinput token predicts probabilities of candidates of the output tokenbased on the token1 105. c₁ through c_(m) denote candidates of an outputtoken. p₁ through p_(m) denote probabilities of the candidates. Based onthe probabilities, an output token is determined from the candidates.For example, a token having a highest probability is selected from thecandidates to be the output token.

A conditional probability of a token t_(i) at a step i may be expressedas shown in Equation 1 below, for example.

p(t_(i)|t₁, t₂, . . . , t_(i−1), He)   (1)

In Equation 1, H_(e) denotes a hidden representation of the encoder andcorresponds to the encoded feature 110. t₁ through t_(i−1) denote inputtokens selected so far. For ease of description, although FIG. 1illustrates that the encoded feature 110 is applied to a node of a firststep in the decoder 120, the encoded feature 110 may also be similarlyapplied to other nodes thereafter.

Typically, in a sequence-to-sequence model which performsauto-regressive decoding, a token selected as an output token isdetermined to be a next input token. Since the output token isdetermined depending on input tokens selected up to the previous step,when the next input token is selected as the output token in thesequence-to-sequence model, a performance of predicting a next token maybe degraded due to the selected output token.

For example, among the candidates c₁ through c_(m) of the output token,c_(i) having a lower probability may be determined to be the outputtoken. Due to a characteristic of generating the subsequent output basedon previous tokens, when a next token is to be generated based on tokenshaving relatively low probabilities in such a typical process ofauto-regressive decoding, it may be difficult to efficiently generatesubsequent tokens if probabilities of previous tokens are relativelylow. For example, although c_(i) has a highest probability,probabilities of c₁ through c_(m) may be low overall. As an example, inan environment in which model ensemble auto-regressive decoding isperformed by selecting and using one of tokens generated in variousartificial neural network models, a next token may be generated based ona token generated by another artificial neural network as an ensembleresult.

Thus, when a probability of an input token is relatively low, anartificial neural network may not accurately predict t corresponding toan output token based on the relation

p(t_(i)|t₁, t₂, . . . , t_(i−1), H_(e))

In an example, such an issue may be solved if a probability iscalculated by excluding the token in Equation 1 related to theconditional probability in order to accurately predict the output tokent_(i) based on the input token t_(i−1) having the lower probability.However, since the conditional probability may be learned from learningdata, if the conditional probability is calculated independently of theprevious token, an original goal of the neural network may not beachieved or accurately substantially lowered.

In an example of the decoding method herein, a special token t_(NC) thatdoes not affect the conditional probability may be used to accuratelypredict the output token t_(i) even when the probability of the inputtoken t_(i−1) is low.

Referring to FIG. 1, a probability p_(i) of the output token c_(i)determined as an output for the token1 105 is compared with a thresholdprobability θ_(n). The threshold probability θ_(nc) may be apredetermined value, for example, a minimum value of an input tokenprobability by which a desired output is to be obtained. For example, itmay be determined that a desired output may be obtained when theprobability p_(i) of the output token c_(i) is higher than the thresholdprobability θ_(nc). In this example, the output token c_(i) isdetermined to be a next input token.

If the probability p_(i) of the output token c_(i) is lower than thethreshold probability θ_(nc), it may be determined that a desired outputmay not be obtained when the output token c_(i) is determined to be anext input token. Thus, if the probability p_(i) of the output tokenc_(i) is lower than the threshold probability θ_(nc), the output tokenc_(i) may not be determined or used as a next input token. In thisexample, a pre-defined special token NC is determined to be, or used as,the next input token instead of the output token c_(i).

A reliability of candidates of a next output token predicted based on aspecial token may be higher than a reliability of candidates of a nextoutput token predicted based on an output token having a probabilitylower than a threshold probability. The artificial neural network may bepre-trained to satisfy such characteristics. A method of training theartificial neural network will be described in detail later. When theprobability p_(i) of the output token c_(i) is lower than the thresholdprobability θ_(nc), a reliability of candidates of a next output tokenpredicted based on the special token NC may be higher than a reliabilityof candidates of a next output token predicted based on c_(i).

A reliability may refer to a reliability of a token that is a degreecloser to a correct answer token with regard to the learning data. Whenthe probability p_(i) of the output token c_(i) is lower than thethreshold probability θ_(nc), an output token having a highestprobability among the candidates of the next output token predictedbased on the special token NC may be closer to the correct answer tokenin comparison to an output token having a highest probability among thecandidates of the next output token predicted based on c_(i).

When the artificial neural network determines a next input token byselecting one of the output token and the special token, a next outputtoken may be determined based on at least one input token and the nextinput token. When the special token NC is determined as the next inputtoken, the artificial neural network determines a token3 125corresponding to the next output token based on the token1 105corresponding to the input token and the special token NC correspondingto the next input token.

FIG. 2 is a diagram illustrating an example of a method of masking aspecial token in an artificial neural network to which an attentionmechanism is applied.

Referring to FIG. 2, a decoder 220 predicts probabilities of candidatesof an output token based on a relationship between at least one inputtoken and the output token. The output token predicts probabilities ofcandidates of an output token using a weight that is selectively variedbased on an input token.

For example, an artificial neural network may be an artificial neuralnetwork to which an attention mechanism is applied. The artificialneural network is trained to appropriately represent a relationship oftokens that are initially predicted based on the attention mechanism.

The decoder 220 masks a special token to prevent other tokens fromattending to the special token such that the special token does notaffect the other tokens.

For example, an output token corresponding to an input token “is” mayattend to a special token NC in addition to the input tokens “Hi”,“this”, and “is”. Because the special token should not affect othertokens, the special token is masked to prevent the output tokencorresponding to the input token “is” from attending to the specialtoken NC.

An encoder-decoder structure artificial neural network system will bedescribed in detail with reference to FIGS. 3 through 5.

FIG. 3 is a block diagram illustrating an example of an encoder-decoderstructure artificial neural network system.

Referring to FIG. 3, an encoder-decoder structure artificial neuralnetwork system includes an artificial neural network 310, which includesan encoder 320 and a decoder 330. The encoder-decoder structureartificial neural network system may also include an input preprocessor340, an attention masking calculator 350, and a token corrector 360. Thedecoder 330 of FIG. 3 may correspond to the decoder 220 described withreference to FIGS. 1 and 2.

The encoder-decoder structure artificial neural network system processesa token having a relatively low probability in the single artificialneural network 310.

A sequence-to-sequence encoder-decoder is a network structure in whichthe encoder 320 and the decoder 330 are integrated and generates asequence of recognition results from an input sequence.

The input preprocessor 340 may perform a pre-processing operation on aninput of an artificial neural network. For example, the inputpreprocessor 340 may remove noise from an input signal or may process aninput signal to be in a form suitable for input to the artificial neuralnetwork, e.g., in a form the artificial neural network is expectingbased on the training of the artificial neural network.

The token corrector 360 may perform a correction operation by replacing,with a special token, an output token of which a probability ispredicted to be less than a threshold probability eNC during executionso as to allow the special token be used for predicting a next inputtoken.

Additionally, the token corrector 360 determines whether the outputtoken is the same as the special token. When the output token is thesame as the special token, a token having a second highest probabilityis determined as the output token. Through this, the special token isprevented from coming out as an output.

The attention masking calculator 350 masks an attention weight toprevent other tokens from attending to the special token such that theother tokens do not have a conditional probability in association withthe special token. The attention masking calculator 350 may operate inthe same manner during learning or training and execution.

FIG. 3 is merely explaining an example of the present disclosure, notingthat additional components are intended in other examples, such as inreconstruction device examples. However, various examples with varioustechnical modifications and variations may be applied based on thedescription of FIG. 3. For example, the decoder 330 may be a broadconcept that includes the attention masking calculator 350 and the tokencorrector 360.

FIG. 4 is a block diagram illustrating an example of an encoder-decoderstructure artificial neural network system using an ensemble technique.

Referring to FIG. 4, an artificial neural network system includes aplurality of artificial neural networks 410, 420, and 430.

In the encoder-decoder structure artificial neural network system usingan ensemble technique, m encoders may not have to correspond to mdecoders. In an example, a specific decoder may not have a correspondingencoder. For example, the artificial neural networks 420 and 430 mayinclude respective decoders 422 and 432, but may not include respectiveencoders 421 and 431.

According to the ensemble technique, an output token may be selectedfrom candidates of the output token based on a combination ofprobabilities of the candidates of the output token predicted by aplurality of decoders 412, 422, and 432 using the plurality ofartificial neural networks 410, 420, and 430.

When the ensemble technique is used, the decoders 412, 422 and 432 maydetermine different output tokens to be a token having a highestprobability. For example, the decoder 412 of the artificial neuralnetwork 410 may determine c1 to be a token having a highest probability,the decoder 422 of the artificial neural network 420 may determine c2 tobe a token having a highest probability, and the decoder 432 of theartificial neural network 430 may determine c3 to be a token having ahighest probability.

Even when each of the decoders 412, 422 and 432 determine differentoutput tokens to be a token having a highest probability, one outputtoken may be determined for all of the decoders. For example, an outputtoken may be selected from candidates of the output token based on acombination of the candidates of the output token predicted by theplurality of decoders 412, 422, and 432.

When the selected output token is forcibly input as a next input tokenof all the decoders, an output token that is not determined to be atoken having a highest probability may be input as a next input token insome of the decoders.

For example, c1 may be selected as an output token based on acombination of probabilities of candidates c1, c2, and c3 of the outputtoken predicted by the plurality of decoders 412, 422, and 432. In thisexample, the decoder 422 and the decoder 423 may generate a next tokenbased on c1 which is the candidate of the output token generated by thedecoder 412 instead of generating a next output token based on thecandidates c2 and c3 determined as tokens having a highest probabilityby the decoder 422 and the decoder 423. Due to the auto-regressivecharacteristic of obtaining a next token based on previous tokens, theartificial neural networks 420 and 430 using the next input token c1having a relatively lower probability as an input in the decoders 422and 423 thereof may not accurately predict a next output token.

The encoder-decoder structure artificial neural network system using theensemble technique may use a special token that does not affect theconditional probability so as to normally predict an output token using,as an input, a token that is not determined as a token having thehighest probability by the encoder-decoder structure artificial neuralnetwork system.

When implementing an ensemble of numerous artificial neural networks, aprobability P_(j)(t_(max)) of t_(max) determined as the output token foreach j^(th) artificial neural network is compared to a probability ofthe artificial neural network 410 having determined that t_(max) has thehighest probability. For example, among the candidates c1, c2, and c3 ofthe output token, a probability p2(c1) of c1 determined as the outputtoken in the artificial neural network 420 may be compared to aprobability p1(c1) of c1 in the artificial neural network 410 havingdetermined that c1 has the highest probability.

When an output token probability difference

${\max\limits_{1 \leq k \leq m}\left( {P_{k}\left( t_{{ma}\; x} \right)} \right)} - P_{j\;}$

between the artificial neural network 410 having determined that theoutput token probability is the highest and a j^(th) artificial neuralnetwork is greater than a threshold, the special token is selected to bea next input token of the j^(th) artificial neural network. For example,when a difference between p1(c1) and p2(c1) is greater than a thresholdθ_(DIFF), the special token may be selected to be the next input tokenof the artificial neural network 420 instead of c1. When the outputtoken probability difference between the artificial neural network 410having determined that the output token probability is the highest andthe j^(th) artificial neural network is less than the threshold, thedetermined output token may be selected to be the next input token ofthe j^(th) artificial neural network.

When the output token probability difference

${\max\limits_{1 \leq k \leq m}\left( {P_{k}\left( t_{{ma}\; x} \right)} \right)} - P_{j}$

between the artificial neural network having determined that the outputtoken probability is the highest and the j^(th) artificial neuralnetwork is greater than the threshold, a reliability of candidates of anext output token predicted based on the special token is higher than areliability of candidates of a next output token predicted based on theoutput token.

The description of the method of masking the special token in theexample of FIG. 2 may also be applied to the encoder-decoder structureartificial neural network system using the ensemble technique of FIG. 4,for example.

A token corrector 460 performs correction by replacing, with a specialtoken, an output token of which a probability is predicted to be lessthan a threshold probability θ_(NC) during execution so that the specialtoken may be used to predict a next input token.

During the execution, the token corrector 460 selects the special tokenas a next input token of a j^(th) artificial neural network when theoutput token probability difference

${\max\limits_{1 \leq k \leq m}\left( {P_{k}\left( t_{{ma}\; x} \right)} \right)} - P_{j}$

between the artificial neural network having determined that the outputtoken probability is the highest and the j^(th) artificial neuralnetwork is greater than the threshold.

Additionally, the token corrector 460 determines whether the outputtoken is the same as the special token. When the output token is thesame as the special token, a token having a second highest probabilityis determined as the output token. Accordingly, the special token may beprevented from coming out, or provided as an output.

An attention masking calculator 450 masks an attention weight to preventother tokens from attending to the special token such that the othertokens do not have a conditional probability in association with thespecial token. The attention masking calculator 450 may operate in thesame manner during learning as well as during execution, inference, orimplementation.

FIG. 5 is a block diagram illustrating an example of a speechrecognition artificial neural network having an additional languagemodel artificial neural network. The speech recognition artificialneural network may be implemented in an example, as a speech recognitionapparatus. The speech recognition apparatus may collect sound or aninput from a user using a receiver or sensor (e.g., a microphone) thatis a component of the speech recognition apparatus, or may receive thespeech signal from the receiver, wherein the receiver is separate orexternal to the speech recognition apparatus.

The speech recognition apparatus may be an electronic device used byindividual users and may be or include, for example, a smart phone, asmart pad, a wearable device such as a smart band, a personal digitalassistant (PDA), a laptop, an internal component thereof, or astandalone apparatus in connection thereto. In another example, thespeech recognition apparatus is an electronic device shared by aplurality of users and includes, for example, a speech recognitionspeaker and a speech recognition TV.

Referring to FIG. 5, a speech recognition artificial neural network,including a language model artificial neural network, includes a speechrecognition model artificial neural network 510 and a language modelartificial neural network 520.

In an example of speech recognition, i.e., machine speech recognition, adecoder 512 may output a sequence of words corresponding to arecognition result using speech or a speech signal as an input of anencoder 511. Additionally, the language model artificial neural network520 may include a decoder 521 which determines a probability of thesequence of words in a general sentence. The language model artificialneural network 520 may be used to improve a performance.

A speech recognition model artificial neural network, which predicts orestimates a word based on an actual speech at every point in time, mayproduce a word by listening to the speech and combining phonemesaccording to a pronunciation even if the word is a new word. However, inan example of a language model artificial neural network that has beentrained using general sentences, an output probability of the word maybe very low when the word is a new word (for example, a new place, anewly created word, and the like). When trying to predict a next wordusing a word having such a low probability as an input, the languagemodel artificial neural network may not make a correct predictionbecause the next word is to be output from an input that has not beenlearned.

A token corrector 560 and an attention masking calculator 550 may beused to introduce a special token that does not affect the conditionalprobability described with reference to FIG. 4 to the language modelartificial neural network 520. Accordingly, the language modelartificial neural network 520 may normally predict an output token usinga token that is not determined as a token having a highest probabilityby the language model artificial neural network 520 as an input.

The speech signal that is input to the encoder 511 may refer to ananalog wave form captured or input to the speech recognition apparatus,that is then converted into a digital waveform, and may include theaforementioned noise reduction, and in some examples, then convertedinto feature data for the digital waveform prior to being acted on orapplied/provided to the speech recognition model 510, and/or may referto such feature data in the example speech sequence format that is actedon or applied/provided to the speech recognition model 510. Thus, forease of description and not to limit examples thereto, hereinafter forthe Specification the speech signal term will be discussed ascorresponding to such post-collection processing having been performedon captured audio to ultimately generate the example feature data in theexample speech sequence form, for application/provision/input to thespeech recognition model, i.e., in the form the speech recognition modelexpects such application/provision/input of information. As noted above,the speech recognition apparatus may perform all such post-collectionprocessing of the captured speech and itself implement the speechrecognition model 510, or the speech recognition apparatus may performnone, some, or all such post-collection processing of the capturedspeech, while a speech recognition server (as another speech recognitionapparatus example) may then perform any remaining post-collectionprocessing for the captured speech to generate corresponding speechinformation in the form expected by the speech recognition model 510,and implement the speech recognition model 510, e.g., by way of thespeech recognition apparatus.

An example in a speech recognition artificial neural network having anadditional language model artificial neural network will be described indetail with reference to FIGS. 6 through 7B.

FIG. 6 is a diagram illustrating an example of a speech recognitionartificial neural network system which includes an additional languagemodel artificial neural network.

Referring to FIG. 6, a speech recognition artificial neural networksystem including an additional language model artificial neural networkincludes an encoder 610, a speech recognition artificial neural networkdecoder 620, and a language model artificial neural network decoder 630.

The speech recognition artificial neural network system including anadditional language model artificial neural network may extract a speechfeature 640 from an input speech. The input speech is a speech signalincluding information for each of a plurality of frames. The speechfeature 640 is a sequence of information extracted in units of at leastone frame and represented by a multidimensional vector. Hereinafter, anexample in which the input speech is “Hi MinChul this is” will bedescribed for ease of description.

The speech recognition artificial neural network system including anadditional language model artificial neural network may generate arecognition result sequence from an input speech sequence using anensemble of the language model artificial neural network decoder 630 andthe speech recognition artificial neural network decoder 620. “Ensemble”may refer to the individual neural network modules taken or consideredtogether. For example, the combination of the language model artificialneural network decoder 630 and the speech recognition artificial neuralnetwork decoder 620.

The language model artificial neural network decoder 630 and the speechrecognition artificial neural network decoder 620 output recognitionresults in units of tokens, and generate a final recognition result byensembling the recognition results based on an ensemble weight.

For example, the speech recognition artificial neural network decoder620 determines candidates of an output token based on the input speechand a recognition result determined in advance. Also, the language modelartificial neural network decoder 630 determines candidates of an outputtoken based on a recognition result determined in advance. In thisexample, the candidates of each of the output tokens may be ensembledbased on an ensemble weight, so that a final recognition result isgenerated.

The encoder 610 and the decoder 620 are previously trained to generate arecognition result sequence from a sequence of correct answer text pairscorresponding to the input speech. Also, the language model artificialneural network decoder 630 is previously trained to generate arecognition result sequence from a predetermined text sequence.

In an example, encoder 610 encodes the speech feature 640 to generate anencoded feature 650. The encoder 610 generates encoded information bychanging a dimension of the speech feature 640. The encoded feature 650is applied to the speech recognition artificial neural network decoder620. The speech recognition artificial neural network decoder 620generates candidates of an output token based on the encoded feature 650and the previously determined recognition result in units of tokens.Also, the language model artificial neural network decoder 630 generatescandidates of an output token based on the previously determinedrecognition result in units of tokens. The two recognition results areensembled based on a predetermined ensemble weight, so that a finalrecognition result is generated. For example, an ensemble weight of thespeech recognition artificial neural network decoder 620 and thelanguage model artificial neural network decoder 630 may be 1:0.2.

An example of an application of an ensemble technique in a languagemodel artificial neural network decoder and a speech recognitionartificial neural network decoder will be described in detail withreference to FIGS. 7A and 7B.

FIG. 7A is a diagram illustrating an example of an application of anensemble technique in a language model artificial neural network decoderand a speech recognition artificial neural network decoder.

Referring to FIG. 7A, a speech recognition artificial neural networkdecoder 710 and a language model artificial neural network decoder 720may respectively correspond to the speech recognition artificial neuralnetwork decoder 620 and the language model artificial neural networkdecoder 630 of FIG. 6, as a non-limiting example.

The speech recognition artificial neural network decoder 710 and thelanguage model artificial neural network decoder 720 may each be anauto-regressive decoder and predict probabilities of candidates of anoutput token based on an input token “Hi”. For example, the speechrecognition artificial neural network decoder 710 outputs “MinChul” and“Bixby” as candidates of an output token based on the input token “Hi”and an encoded speech feature and predicts probabilities of thecandidates to be 0.7 and 0.1, respectively. Also, the language modelartificial neural network decoder 720 outputs “MinChul” and “Bixby” ascandidates of an output token based on the input token “Hi” and predictsprobabilities of the candidates to be 0.001 and 0.8, respectively.

The speech recognition artificial neural network decoder 710 and thelanguage model artificial neural network decoder 720 determines anoutput token among the candidates based on an ensemble weight. Forexample, “MinChul” has a final weight of 0.7002 (=0.7+0.2*0.001) and“Bixby” has a final weight of 0.26 (=0.1+0.2*0.8). In this example,“MinChul” is determined to be the output token.

In an auto-regressive decoding environment, the language modelartificial neural network decoder 720 inputs “MinChul” as a next inputtoken determined as an ensemble result instead of “Bixby” which isdetermined by the language model artificial neural network decoder 720as a token having a highest probability. Because “MinChul” is determinedto have a lower probability by the language model artificial neuralnetwork decoder 720, it may be difficult to accurately predict a nextoutput token in the language model artificial neural network decoder720.

For example, the language model artificial neural network decoder 720outputs “this” and “dis” as candidates of a next output token based onthe next input token “MinChul” and predicts probabilities of thecandidates to be 0.01 and 0.01, respectively. The speech recognitionartificial neural network decoder 710 outputs “this” and “dis” ascandidates of a next output token based on the next input token“MinChul” and predicts probabilities of the candidates “this” and “dis”to be 0.34 and 0.38, respectively. Because pronunciations of “this” and“dis” are similar, the probabilities of “this” and “dis” may be similarin the speech recognition artificial neural network decoder 710.

The speech recognition artificial neural network decoder 710 and thelanguage model artificial neural network decoder 720 determine a nextoutput token among the candidates based on an ensemble weight. Forexample, “this” has a final weight of 0.342 (=0.34+0.2*0.01) and “dis”has a final weight of 0.382 (=0.38+0.2*0.01). Thus, “dis” is determinedto be the next output token. As such, the speech recognition artificialneural network decoder 710 and the language model artificial neuralnetwork decoder 720 may output “dis”, which is different from the inputspeech “this”.

In the foregoing example, the probability of “MinChul” determined by anensemble of the speech recognition artificial neural network decoder 710and the language model artificial neural network decoder 720 amongcandidates of a current output token predicted by the language modelartificial neural network decoder 720 may be lower than a thresholdprobability. Due to the lower probability of “MinChul”, the languagemodel artificial neural network decoder 720 may not accuratelydistinguish between “this” and “dis” as the next output token. As aperformance of the language model artificial neural network decoder 720which complements a performance of the speech recognition artificialneural network decoder 710 is degraded, an incorrect result may beobtained, such as outputting “this” which is different from “dis” asdescribed above.

An example of an application of an ensemble technique in a languagemodel artificial neural network decoder and a speech recognitionartificial neural network decoder using a special token will be furtherdescribed with reference to FIG. 7B.

FIG. 7B is a diagram illustrating an example of an application of anensemble technique in a language model artificial neural network decoderand a speech recognition artificial neural network decoder based on theimplementation of a special token.

Referring to FIG. 7B, the speech recognition artificial neural networkdecoder 710 and the language model artificial neural network decoder 720determine “MinChul” among candidates to be an output token using anensemble weight.

The language model artificial neural network decoder 720 compares aprobability of “MinChul” and a threshold probability determined inadvance. The threshold probability is, for example, 0.01. As illustratedin FIG. 7B, the probability of “MinChul” predicted by the language modelartificial neural network decoder 720 is 0.001, which is less than thethreshold probability of 0.01. Thus, the language model artificialneural network decoder 720 selects a special token NC as a next inputtoken instead of “MinChul”.

The language model artificial neural network decoder 720 outputs “this”and “dis” as candidates of a next output token based on the specialtoken NC corresponding to the next input token and predictsprobabilities of the candidates “this” and “dis” to be 0.4 and 0.1,respectively. The speech recognition artificial neural network decoder710 outputs “this” and “dis” as candidates of a next output token basedon the special token NC corresponding to the next input token andpredicts probabilities of the candidates “this” and “dis” to be 0.34 and0.38, respectively.

The speech recognition artificial neural network decoder 710 and thelanguage model artificial neural network decoder 720 determine a nextoutput token among the candidates using an ensemble weight. For example,“this” has a final weight of 0.42 (=0.34+0.2*0.4) and “dis” has a finalweight of 0.4 (=0.38+0.2*0.1). Thus, “this” is determined to be the nextoutput token.

FIG. 8 is a flowchart illustrating an example of a decoding method in anartificial neural network. The operations in FIG. 8 may be performed inthe sequence and manner as shown, although the order of some operationsmay be changed or some of the operations omitted without departing fromthe spirit and scope of the illustrative examples described. Many of theoperations shown in FIG. 8 may be performed in parallel or concurrently.One or more blocks of FIG. 8, and combinations of the blocks, can beimplemented by special purpose hardware-based computer that perform thespecified functions, or combinations of special purpose hardware andcomputer instructions. In addition to the description of FIG. 8 below,the descriptions of FIGS. 1-7 are also applicable to FIG. 8, and areincorporated herein by reference. Thus, the above description may not berepeated here.

Referring to FIG. 8, operations 810 through 830 may be performed by thedecoder 330 of FIG. 3. The decoder 330 may be implemented by one or morehardware components or one or more components including both hardwareand software.

In operation 810, the decoder 330 predicts probabilities of candidatesof an output token based on at least one input token that was initiallyinput.

In operation 820, the decoder 330 determines an output token from thecandidates based on the determined probabilities. For example, thedecoder 330 may select a token with a highest probability from thecandidates. Also, the decoder 330 may also select the output token basedon a combination of probabilities of candidates predicted by a secondartificial neural network interworking with the first artificial neuralnetwork and the probabilities of the candidates predicted by the firstartificial neural network.

In operation 830, the decoder 330 determines a next input token byselecting one of the output token and a pre-defined special token basedon the determined probability of the output token.

The at least one input token includes at least one of a special tokenand an output token previously predicted by the artificial neuralnetwork.

An input layer of the artificial neural network may include a pluralityof nodes corresponding to the candidates and a special nodecorresponding to the special token.

FIG. 9 is a flowchart illustrating an example of a method of processinga token with a low probability in an auto-regressive-based sequencegenerating artificial neural network. The operations in FIG. 9 may beperformed in the sequence and manner as shown, although the order ofsome operations may be changed or some of the operations omitted withoutdeparting from the spirit and scope of the illustrative examplesdescribed. Many of the operations shown in FIG. 9 may be performed inparallel or concurrently. One or more blocks of FIG. 9, and combinationsof the blocks, can be implemented by special purpose hardware-basedcomputer that perform the specified functions, or combinations ofspecial purpose hardware and computer instructions. In addition to thedescription of FIG. 9 below, the descriptions of FIGS. 1-8 are alsoapplicable to FIG. 9, and are incorporated herein by reference. Thus,the above description may not be repeated here.

Referring to FIG. 9, in operation 901, an auto-regressive-based sequencegenerating artificial neural network receives an input. For example, aspeech recognition artificial neural network system may receive a speechinput.

In operation 902, an encoder generates an encoded feature from thereceived input. For example, the encoder may generate encodedinformation by changing a dimension of a speech feature.

In operation 903, a decoder predicts probabilities of candidates of anoutput token based on the encoded feature. The decoder obtains theoutput token based on information calculated by the encoder for eachstep. In this example, the decoder obtains the output token depending oninput tokens selected up to a previous step.

In operation 904, the decoder determines the output token fromcandidates based on the predicted probabilities. For example, thedecoder determines t_(i) having a highest probability at an i^(th) stepto be the output token.

In operation 905, the decoder determines whether the output token is thesame as a special token.

In operation 906, when the output token is the same as the specialtoken, the decoder determines a token having a second highestprobability among the candidates to be the output token. Through this,the special token is prevented from being output.

In operation 907, when the output token is different from the specialtoken, the decoder compares a probability of the output token to athreshold probability.

In operation 908, when the probability of the output token is higherthan the threshold probability, the decoder selects the output token tobe a next input token.

In operation 909, when the probability of the output token is less thanthe threshold probability, the decoder selects the special token as thenext input token instead of the output token.

In operation 910, the decoder masks the special token to prevent othertokens from attending to the special token such that the special tokendoes not affect the other tokens.

A special token t_(NC) that does not affect a conditional probability isintroduced such that t_(i) which is an output token is adequately, e.g.,within a predetermined accuracy, predicted even when a probability of aninput token t_(i−1) is low. The special token is learned to havecharacteristics as shown in Equation 2 below.

p(t _(i) |<T ₁ >, t _(NC) , <T ₂ >, H _(e))=p(t _(i) |<T ₁ >, <T ₂ >, H_(e)) for any t _(i)   (2)

In Equation 2, <T₁>, <T₂> is any token sequence and includes an emptysequence. When t_(NC) is learned according to Equation 2, t_(i−1) isreplaced with t_(NC) to prevent a conditional probability including thetoken t_(i−1) from being calculated if a probability of t_(i−1) is lessthan a threshold θ_(NC), so that a next token is correctly predicted. Anartificial neural network is trained by changing tokens of a correctanswer sentence used for learning at a predetermined probability tot_(NC) at a probability θ_(Replace).

When main and auxiliary artificial neural networks are distinguished,the training may be performed by changing a correct answer token at theprobability θ_(Replace) in learning data of an auxiliary artificialneural network. For example, in a case of a speech recognitionartificial neural network system having an additional language modelartificial neural network, the language model artificial neural networkmay be the auxiliary artificial neural network. In this example, thetraining is performed by changing a correct answer token at theprobability θ_(Replace) in learning data of the language modelartificial neural network.

FIG. 10 is a block diagram illustrating an apparatus for processing atoken with a low probability in an auto-regressive-based sequencegenerating artificial neural network.

Referring to FIG. 10, an apparatus 1000 for processing a token with alow probability in an auto-regressive-based sequence generatingartificial neural network includes one or more sensors 1010, a processor1030, and a communication interface 1070. The apparatus 1000 furtherincludes a memory 1050 and a display 1090. The one or more sensors 1010,the processor 1030, the memory 1050, the communication interface 1070,and the display 1090 may communicate with each other through acommunication bus 1005.

The one or more sensors 1010 include, for example, a microphone sensorand a voice sensor, but are not so limited.

The processor 1030 performs any one, any combination, or all operationsor methods described with reference to FIGS. 1 through 9, or analgorithm corresponding to the at least one method. The processor 1030executes a program and controls the apparatus 1000. The computerinstructions or code executed by the processor 1030 are stored in thememory 1050.

The processor 1030 includes, for example, a central processing unit(CPU) or a graphics processing unit (GPU).

The memory 1050 stores data processed by the processor 1030. Forexample, the memory 1050 stores a program. The memory 1050 may be avolatile memory or a non-volatile memory.

The communication interface 1070 is connected to the sensor(s) 1010, theprocessor 1030, and the memory 1050 to perform data transmission andreception. The communication interface 1070 is connected to an externaldevice to perform data transmission and reception. In the followingdescription, an expression “transmitting and receiving “A”” refers totransmitting and receiving data or information representing “A”.

The communication interface 1070 is implemented as, for example, acircuitry in the apparatus 1000. In an example, the communicationinterface 1070 may include an internal bus and an external bus. Inanother example, the communication interface 1070 may be an elementconfigured to connect the apparatus 1000 to an external device. Thecommunication interface 1070 receives data from the external device andtransmits the data to the processor 1030 and the memory 1050.

The display 1090 displays a decoding result. For example, a translationresult and a speech recognition result may be displayed on the display1090.

The apparatuses, input preprocessor 340/440/441/442, attention maskingcalculator 350/450/550, encoder 320/411/421/431/511, decoder330/412/422/432/512/521, token corrector 360/460/560, speechpreprocessor 540, sensor, 1010, memory 1050, processor 1030, display1090, communication interface 1070, components, devices, and othercomponents described herein with respect to FIGS. 1-10 are, and areimplemented by, hardware components. Examples of hardware componentsthat may be used to perform the operations described in this applicationwhere appropriate include controllers, sensors, generators, drivers,memories, comparators, arithmetic logic units, adders, subtractors,multipliers, dividers, integrators, and any other electronic componentsconfigured to perform the operations described in this application. Inother examples, one or more of the hardware components that perform theoperations described in this application are implemented by computinghardware, for example, by one or more processors or computers. Aprocessor or computer may be implemented by one or more processingelements, such as an array of logic gates, a controller and anarithmetic logic unit, a digital signal processor, a microcomputer, aprogrammable logic controller, a field-programmable gate array, aprogrammable logic array, a microprocessor, or any other device orcombination of devices that is configured to respond to and executeinstructions in a defined manner to achieve a desired result. In oneexample, a processor or computer includes, or is connected to, one ormore memories storing instructions or software that are executed by theprocessor or computer. Hardware components implemented by a processor orcomputer may execute instructions or software, such as an operatingsystem (OS) and one or more software applications that run on the OS, toperform the operations described in this application. The hardwarecomponents may also access, manipulate, process, create, and store datain response to execution of the instructions or software. Forsimplicity, the singular term “processor” or “computer” may be used inthe description of the examples described in this application, but inother examples multiple processors or computers may be used, or aprocessor or computer may include multiple processing elements, ormultiple types of processing elements, or both. For example, a singlehardware component or two or more hardware components may be implementedby a single processor, or two or more processors, or a processor and acontroller. One or more hardware components may be implemented by one ormore processors, or a processor and a controller, and one or more otherhardware components may be implemented by one or more other processors,or another processor and another controller. One or more processors, ora processor and a controller, may implement a single hardware component,or two or more hardware components. A hardware component may have anyone or more of different processing configurations, examples of whichinclude a single processor, independent processors, parallel processors,single-instruction single-data (SISD) multiprocessing,single-instruction multiple-data (SIMD) multiprocessing,multiple-instruction single-data (MISD) multiprocessing, andmultiple-instruction multiple-data (MIMD) multiprocessing.

The methods illustrated and discussed with respect to FIGS. 1-10, andthat perform the operations described in this application, are performedby computing hardware, for example, by one or more processors orcomputers, implemented as described above executing instructions orsoftware to perform the operations described in this application thatare performed by the methods. For example, a single operation or two ormore operations may be performed by a single processor, or two or moreprocessors, or a processor and a controller. One or more operations maybe performed by one or more processors, or a processor and a controller,and one or more other operations may be performed by one or more otherprocessors, or another processor and another controller. One or moreprocessors, or a processor and a controller, may perform a singleoperation, or two or more operations.

Instructions or software to control computing hardware, for example, oneor more processors or computers, to implement the hardware componentsand perform the methods as described above may be written as computerprograms, code segments, instructions or any combination thereof, forindividually or collectively instructing or configuring the one or moreprocessors or computers to operate as a machine or special-purposecomputer to perform the operations that are performed by the hardwarecomponents and the methods as described above. In one example, theinstructions or software include machine code that is directly executedby the one or more processors or computers, such as machine codeproduced by a compiler. In another example, the instructions or softwareincludes higher-level code that is executed by the one or moreprocessors or computers using an interpreter. The instructions orsoftware may be written using any programming language based on theblock diagrams and the flow charts illustrated in the drawings and thecorresponding descriptions in the specification, which disclosealgorithms for performing the operations that are performed by thehardware components and the methods as described above.

The instructions or software to control computing hardware, for example,one or more processors or computers, to implement the hardwarecomponents and perform the methods as described above, and anyassociated data, data files, and data structures, may be recorded,stored, or fixed in or on one or more non-transitory computer-readablestorage media. Examples of a non-transitory computer-readable storagemedium include read-only memory (ROM), random-access programmable readonly memory (PROM), electrically erasable programmable read-only memory(EEPROM), random-access memory (RAM), dynamic random access memory(DRAM), static random access memory (SRAM), flash memory, non-volatilememory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs,DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, asnon-limiting blue-ray or optical disk storage examples, hard disk drive(HDD), solid state drive (SSD), flash memory, a card type memory such asmultimedia card micro or a card (for example, secure digital (SD) orextreme digital (XD)), magnetic tapes, floppy disks, magneto-opticaldata storage devices, optical data storage devices, hard disks,solid-state disks, and any other device that is configured to store theinstructions or software and any associated data, data files, and datastructures in a non-transitory manner and provide the instructions orsoftware and any associated data, data files, and data structures to oneor more processors or computers so that the one or more processors orcomputers can execute the instructions. In one example, the instructionsor software and any associated data, data files, and data structures aredistributed over network-coupled computer systems so that theinstructions and software and any associated data, data files, and datastructures are stored, accessed, and executed in a distributed fashionby the one or more processors or computers.

While this disclosure includes specific examples, it will be apparentafter an understanding of the disclosure of this application thatvarious changes in form and details may be made in these exampleswithout departing from the spirit and scope of the claims and theirequivalents. The examples described herein are to be considered in adescriptive sense only, and not for purposes of limitation. Descriptionsof features or aspects in each example are to be considered as beingapplicable to similar features or aspects in other examples. Suitableresults may be achieved if the described techniques are performed in adifferent order, and/or if components in a described system,architecture, device, or circuit are combined in a different manner,and/or replaced or supplemented by other components or theirequivalents. Therefore, the scope of the disclosure is defined not bythe detailed description, but by the claims and their equivalents, andall variations within the scope of the claims and their equivalents areto be construed as being included in the disclosure.

What is claimed is:
 1. A processor-implemented decoding method in afirst neural network, the decoding method comprising: predictingprobabilities of candidates of an output token based on at least onepreviously input token; determining the output token among thecandidates based on the predicted probabilities; and determining a nextinput token by selecting one of the output token and a pre-definedspecial token based on a determined probability of the output token. 2.The method of claim 1, wherein when the special token is determined tobe the next input token, a reliability of candidates of a next outputtoken predicted based on the special token is higher than a reliabilityof candidates of a next output token predicted based on the outputtoken.
 3. The method of claim 2, wherein the determining of the outputtoken comprises: selecting the output token from the candidates based ona combination of probabilities of the candidates predicted by one ormore second neural networks interworking with the first artificialneural network, and probabilities of the candidates predicted by thefirst neural network.
 4. The method of claim 1, wherein an input layerof the neural network comprises a plurality of nodes corresponding toeach of the candidates, and a special node corresponding to the specialtoken.
 5. The method of claim 1, wherein the at least one previouslyinput token comprises at least one of an output token previouslypredicted by the neural network and the special token.
 6. The method ofclaim 1, wherein the determining of the output token comprises:selecting a token having a highest probability among the candidates. 7.The method of claim 3, wherein the method is a recognition method usingthe first and the one or more second neural networks.
 8. The method ofclaim 1, wherein the determining of the next input token comprises:comparing a probability of the output token to a threshold probability;selecting the special token as the next input token when the probabilityof the output token is lower than the threshold probability; andselecting the output token as the next input token when the probabilityof the output token is higher than or equal to the thresholdprobability.
 9. The method of claim 1, wherein the predicting of theprobabilities of the candidates of the output token comprises:predicting probabilities of the candidates of the output token based ona relationship between the at least one previously input token and theoutput token.
 10. The method of claim 1, further comprising: determininga next output token based on the at least one previously input token andthe next input token.
 11. The method of claim 2, further comprising:masking the special token to prevent tokens other than the special tokenfrom attending to the special token.
 12. The method of claim 1, furthercomprising: setting a relationship between the next output token and thespecial token to be less than or equal to a predetermined value when thespecial token is determined to be the next input token.
 13. The methodof claim 1, wherein the determining of the output token comprises:determining whether the output token is similar to the special token;and determining a token having a second highest probability among thecandidates to be the output token when the output token is similar tothe special token.
 14. A processor-implemented decoding method in afirst neural network, the method comprising: predicting probabilities ofcandidates of an output token based on at least one previously inputtoken; selecting the output token from the candidates based on acombination of probabilities of candidates predicted by one or moresecond neural networks interworking with the first neural network andprobabilities of candidates predicted by the first neural network;comparing a probability of the output token in the first neural networkand a probability of the output token in a determined neural networkamong the one or more second neural networks which has a highestprobability of the output token; and determining a next input token inthe first neural network by selecting one of the output token and apre-defined special token based on a result of the comparing.
 15. Themethod of claim 14, wherein the determining of the next input tokencomprises: selecting the special token to be the next input token of thefirst neural network when a difference between the probability of theoutput token in the first neural network and the probability of theoutput token in the determined neural network is greater than or equalto a threshold value; and selecting the output token to be the nextinput token of the first neural network when a difference between theprobability of the output token in the first neural network and theprobability of the output token in the determined neural network is lessthan the threshold value.
 16. The method of claim 15, wherein when thedifference between the probability of the output token in the firstneural network and the probability of the output token in the determinedneural network is greater than or equal to the threshold value, areliability of candidates of a next output token predicted based on thespecial token is higher than a reliability of candidates of a nextoutput token predicted by the output token.
 17. The method of claim 14,wherein an input layer of the neural network comprises a plurality ofnodes corresponding to each of the candidates, and a special nodecorresponding to the special token.
 18. The method of claim 14, whereinthe at least one previously input token comprises at least one of anoutput token previously predicted by the artificial neural network andthe special token.
 19. The method of claim 14, wherein the predicting ofthe probabilities of the candidates of the output token comprises:predicting probabilities of the candidates of the output token based ona relationship between the at least one previously input token and theoutput token.
 20. The method of claim 14, further comprising:determining a next output token based on the at least one previouslyinput token and the next input token.
 21. The method of claim 14,further comprising: masking the special token to prevent tokens otherthan the special token from attending to the special token.
 22. Themethod of claim 14, further comprising: setting a relationship betweenthe next output token and the special token to be less than or equal toa predetermined value when the special token is determined to be thenext input token.
 23. The method of claim 14, wherein the determining ofthe output token comprises: determining whether the output token issimilar to the special token; and determining a token having a secondhighest probability among the candidates to be the output token when theoutput token is similar to the special token.
 24. A non-transitorycomputer-readable storage medium storing instructions that, whenexecuted by a processor, cause the processor to perform the method ofclaim
 1. 25. A speech recognition apparatus comprising: one or moreprocessors configured to implement at least a first neural networkincluding the one or more processors configured to: predictprobabilities of candidates of an output token based on at least onepreviously input token input; determine the output token among thecandidates based on the predicted probabilities; and determine a nextinput token by selecting one of the output token and a pre-definedspecial token based on a determined probability of the output token. 26.The speech recognition apparatus of claim 25, wherein when the specialtoken is determined to be the next input token, a reliability ofcandidates of a next output token predicted based on the special tokenis higher than a reliability of candidates of a next output tokenpredicted based on the output token.
 27. The speech recognitionapparatus of claim 25, wherein an input layer of the neural networkcomprises a plurality of nodes corresponding to each of the candidates,and a special node corresponding to the special token.
 28. The speechrecognition apparatus of claim 25, wherein the at least one previouslyinput token comprises at least one of an output token previouslypredicted by the neural network and the special token.
 29. The speechrecognition apparatus of claim 25, wherein the one or more processorsare configured to select a token having a highest probability among thecandidates.
 30. The speech recognition apparatus of claim 25, whereinthe one or more processors are further configured to implement one ormore second neural networks, and wherein the one or more processors arefurther configured to select the output token from the candidates basedon a combination of probabilities of the candidates predicted by asecond neural network interworking with the first neural network, andprobabilities of the candidates predicted by the first neural network.31. The speech recognition apparatus of claim 25, wherein the one ormore processors are further configured to: compare a probability of theoutput token to a threshold probability; select the special token as thenext input token when the probability of the output token is lower thanthe threshold probability; and select the output token as the next inputtoken when the probability of the output token is higher than or equalto the threshold probability.
 32. The speech recognition apparatus ofclaim 25, wherein the one or more processors are further configured topredict probabilities of the candidates of the output token based on arelationship between the at least one previously input token and theoutput token.
 33. The speech recognition apparatus of claim 25, whereinthe one or more processors are further configured to determine a nextoutput token based on the at least one previously input token and thenext input token.
 34. The speech recognition apparatus of claim 25,wherein the one or more processors are further configured to mask thespecial token to prevent tokens other than the special token fromattending to the special token.
 35. The speech recognition apparatus ofclaim 25, wherein the one or more processors are further configured toset a relationship between the next output token and the special tokento be less than or equal to a predetermined value when the special tokenis determined to be the next input token.
 36. The speech recognitionapparatus of claim 25, wherein the one or more processors are furtherconfigured to: determine whether the output token is similar to thespecial token; and determine a token having a second highest probabilityamong the candidates to be the output token when the output token issimilar to the special token.
 37. A speech recognition apparatus, thespeech recognition apparatus comprising: one or more processorsconfigured to implement at least a first neural network including theprocessor configured to: predict probabilities of candidates of anoutput token based on at least one previously input token; select theoutput token from the candidates based on a combination of probabilitiesof the candidates predicted by one or more second neural networksinterworking with the first neural network and probabilities of thecandidates predicted by the first neural network; compare a probabilityof the output token in the first neural network and a probability of theoutput token in a determined neural network among the one or more secondneural networks which has a highest probability of the output token; anddetermine a next input token in the first neural network by selectingone of the output token and a pre-defined special token based on aresult of the comparing.
 38. The speech recognition apparatus of claim37, wherein the one or more processors are further configured to: selectthe special token to be the next input token of the first neural networkwhen a difference between the probability of the output token in thefirst neural network and the probability of the output token in thedetermined neural network is greater than or equal to a threshold value;and select the output token to be the next input token of the firstneural network when a difference between the probability of the outputtoken in the first neural network and the probability of the outputtoken in the determined neural network is less than the threshold value.39. The speech recognition apparatus of claim 38, wherein when adifference between the probability of the output token in the firstneural network and the probability of the output token in the determinedneural network is greater than or equal to the threshold value, areliability of candidates of a next output token predicted based on thespecial token is higher than a reliability of candidates of a nextoutput token predicted by the output token.
 40. A speech recognitionsystem comprising: one or more processors; one or more memories, storinginstructions that, when executed by the one or more processors,configured the one or more processors to: extract a speech feature froman input speech signal and generate an encoded feature; determineprobabilities of candidates of an output token based on the encodedfeature and a previously determined recognition result; and determine anext output token based on a determined weight of candidates generatedby a language model neural network, and a determined weight ofcandidates determined by a speech recognition neural network.